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The New Grand Challenge 

Complex, global- scale socio-technical 
systems are emerging as computers and 
communications become embedded 
everywhere 

We're coming to depend on the Internet, on 
the payment system, on many others . . . 

How are we to understand them, manage 
them and improve them? 
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Complex Systems 

Since the invention of agriculture and towns about 
10,000 years ago we've been building complex 
systems 

Armies, civil services, religions, industries, 
markets... 

'Long march from status to contract' 
Until recently systems were driven by people - 
using simple incentive mechanisms based on 
personal relationships or physical tokens 



AusCERT, Queensland 
May 18 2011 



Roman Army 






m 


1 









Localised hierarchy (after 100 BC . . .) 
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Chinese Civil Service 





Centralised hierarchy 
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Complex Socio-technical Systems 



Now we have people plus software! 

- The Internet itself 

- The global card payment system 

- The global advertising ecosystem 

- Smart grids for distributing electricity 

- Facebook 



But - with global-scale systems we get conflict 
How do we build such systems to be dependable 
and fit for purpose? 
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Traditional Systems Engineering 

• Build systems for scalability - choose efficient 
algorithms and data structures 

• Once you start to distribute systems, pay attention 
to consistency (file locking, fault tolerance etc) 

• Dump complexity where possible - so put the 
intelligence at the edge of the network 

• See security as 'keeping the bad guys out' by 
adding crypto, authentication, filtering 

• But . . . about 2000, some of us started to realize 
that this is not enough! 
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Economics Matters Too 

Since 2000, we have started to apply economic 
analysis to IT security and dependability 
Systems often fail because the folks who guard them, 
or who could fix them, have insufficient incentives 

- Where banks can dump fraud risk on customers or 
merchants, fraud increases 

- If electricity generation companies don't have an incentive 
to provide reserve capacity, there will be blackouts 

Insecurity is often an 'externality' - a side-effect, like 
environmental pollution 
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IT Economics (1) 

The first distinguishing characteristic of many IT 

product and service markets is network effects 

Metcalfe's law - the value of a network is the 

square of the number of users 

Real networks - phones, fax, email 

Virtual networks - PC architecture versus MAC, 

or Symbian versus WinCE 

Network effects tend to lead to dominant-firm 

markets where the winner takes all 
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IT Economics (2) 

Second common feature of IT product and service 
markets is high fixed costs and low marginal costs 

Competition can drive down prices to marginal 
cost of production 

This can make it hard to recover capital 

investment, unless stopped by patent, brand, 

compatibility . . . 

These effects can also lead to monopoly or 

oligopoly 
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IT Economics (3) 

• Third common feature of IT markets is that 
switching from one product or service to another 
is expensive 

• E.g. switching from Windows to Linux means 
retraining staff, rewriting apps 

• Shapiro- Varian theorem: the net present value of a 
software company is the total switching costs 

• So major effort goes into managing switching 
costs - once you have $3000 worth of songs on a 
$300 iPod, you're locked into iPods 
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IT Economics and Security 

High fixed/low marginal costs, network effects 
and switching costs all tend to lead to dominant- 
firm markets with big first-mover advantage 
So time-to-market is critical 
Microsoft philosophy of 'we'll ship it Tuesday 
and get it right by version 3' is not perverse 
behaviour by Bill Gates but quite rational 
Whichever company had won in the PC OS 
business would have done the same 
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IT Economics and Security (2) 

When building a network monopoly, you must 
appeal to vendors of complementary products 
I.e. app developers for PC versus Apple, Symbian 
versus Palm, Facebook versus Myspace 
Little security in early versions so easier to 
develop apps; win the market; then lock in down 
Payment networks: appeal to merchants first 
Online: choose security technologies that dump 
costs on the user (SSL, not SET) 

How does this affect the infrastructure? 
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The Economics of Dependability (1) 

• In a power cut, a single utility's cost is the lost 
customer minutes 

• The cost to the economy can be much larger! 

• In a grid, who will to provide surplus capacity to 
improve competitors' reliability? 

• Solution: electricity market taxed to pay for 
spinning reserve 

• Many utilities are not as simple because some 
players can dump costs on others 
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Example - Payment Systems 



h 




EMV system 2003- 

5 

Cardholder liable if 

PIN used 

Else merchant pays 

Banks hoped fraud 

would go down 

It went up . . . 
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Why are so many security 
products ineffective? 

Akerlof s Nobel-prize winning paper 'The Market 

for Lemons' introduced asymmetric information 

Suppose a town has 100 used cars for sale: 50 

good ones worth $2000 and 50 lemons worth 

$1000 

What is the equilibrium price of used cars? 
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Why are so many security 
products ineffective? 

• Akerlof s Nobel-prize winning paper The Market 
for Lemons' introduced asymmetric information 

• Suppose a town has 100 used cars for sale: 50 
good ones worth $2000 and 50 lemons worth 
$1000 

• What is the equilibrium price of used cars? 

• If $1500, no good cars will be offered for sale . . . 

• Started the study of asymmetric information 

• Security products are often a 'lemons market' 
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Asymmetric Information (2) 

• Adverse selection - the sick buy more insurance 

• Application to trust: Ben Edelman, 'Adverse 
selection on online trust certifications' (WEIS 06) 

• Websites with a TRUSTe certification are more 
than twice as likely to be malicious 

• In 2006, the top Google ad was about twice as 
likely as the top free search result to be malicious 
(other search engines worse . . .) 

• Ben's conclusion: 'Don't click on ads!' 

• Complex interactions between certification / 
reputation / quality control. . . 

AusCERT, Queensland 
May 18 2011 



Payment System Failure (2008) 




• PEDs 'evaluated under the 
Common Criteria' were 
trivial to tap 

• Weakest-link system 

• GCHQ wouldn't defend 
the brand 

• APACS said (Feb 08) it 
wasn't a problem 

• By July 08 it sure was . . . 
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Payment System Failure (2010) 

• 2009: many believable cardholders said stolen 
cards used in ATMs, PEDs when PIN was safe 

• Oct 2009: 'No-PIN' attack - tell PED it's chip- 
and-pin, tell card it's chip- and- signature 

• Notified industry Dec 09; published Feb, May 
2010; some banks fixed, some still not 

• Dec 2010: banks demanded a student thesis be 
taken down. . . see www.lightbluetouchpaper.org 

• Real cause: spec grew like Topsy, no-one really 
owns it, and liability games issuer / acquirer 
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Agency Effects 



If you are DirNS A and have a nice new hack on 
XP, Vista and Windows 7, do you tell Microsoft? 
Tell - you protect 300m Americans 
Don't tell - you can hack 400m Europeans, 
1000m Chinese,... 

So offence can be favored by governments over 
defense - the 'conflicted mission problem' 
Also: if the Chinese hack US systems, they keep 
quiet. If you hack their systems, you can brag 
about it to the President 
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Resilience of the Internet 

• Inter-X study for ENIS A on the resilience of the 
Internet interconnection ecosystem - with Chris 
Hall, Richard Clayton and ENISA staff 

• Drew on our experience of network business, 
network ops, security engineering, economics 

• Talked to dozens of key players; then draft 
shared with Juniper, Cisco, Huawei, Google, . . . 

• Set out to document what insiders know, and 
collect data too 

• Short version accepted at WEIS 201 1 ; full report 
up at www.ensia.eu and my web page 
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What Could Break the Internet? 

So far, the Internet has survived major incidents 
like 9/11 and Katrina 

But it's ever more critical, both at micro level 
(99% or 99.99%) and macro level (big outages) 
So what could go seriously wrong? 

- Technical infrastructure failure, e.g. power 

- Human infrastructure failure, e.g. flu pandemic 

- Organisational infrastructure failure 

- Bugs, e.g. IPv6 cutover 

- Attacks, e.g. router malware 



AusCERT, Queensland 

May 18 2011 



Internet Resilience (2) 

Failures of the technical or human infrastructure 

are the 'easy' cases to analyze 

Electric power: EMP attack / large solar coronal 

mass ejection leading to regional transmission / 

distribution failure; if too many transformers 

damaged, bad news 

Disruption of society: main planning scenario is 

flu pandemic similar to 1918 (e.g. 2% die, 98% 

stay home for 6 weeks to not get infected!) 
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Internet Resilience (3) 

US transit prices falling 40% p. a.; logical as the 
marginal cost of production is generally zero! 
In 2005-9, Level 3 lost $3.5bn, Global crossing 
$1.4bn, Savvis $175m ... how sustainable is this? 
35,000 Gbit/sec of transit at $3/Mbit/month makes 
the total transit market worth $1.3bn per annum 
Consolidate until survivors get market power? 
(Renesys prediction) 
What happens then? Regulation? By whom? 
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Internet Resilience (4) 

Could Murphy do for the Internet? 

Biggest outage so far was 2008 cable cuts - about 

10 9 1cm - versus Egypt's 10 8 1cm 

IPv6 creates opportunities for bugs . . . 

BGP misconfigurations like route leaks seem to be 

getting rarer and smaller 

Diminishing safety margin might lead to cascade 
failures (BGP tells us reachability not capacity) 
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Internet Resilience (5) 

• This is where it gets hard - the interaction of 
security and economics ! 

• No secure BGP - because ASes can't get local, 
incremental benefit 

• RPKI, route filtering also have poor incentives 

• Will ASes buy spare capacity to help competitors? 

• Asymmetric information: routing tables and much 
else are kept confidential 

• So can't understand topology even to buy 
diversity... 
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Internet Resilience (6) 

How would you bring down the Internet? 

Did China test a 'cyber-nuke' last April or was it 

just an accidental route leak?? 

Might bad guys use a zero-day to get thousands of 

routers to advertise bogus routes and then tear 

them down again? 

If so, how soon could we recover? 

What can we practically do to reduce the scale and 

period of the outage? 
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Possible Ways Forward . . . 

Internet regulation has so far been rather 
unimpressive! 

But we need engagement to teach regulators 
how things work, in case real regulation is 
needed later 

We came up with eleven recommendations 
that have now beenadopted as ENISA 
policy 
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Recommendations 

1: Independent body to investigate majopr 
incidents and report on them publicly 

2: Collect consistent, comprehensive, long- 
term network performance data 

3: Research better metrics for resilience of 
complex multi-layer networks 

4: Develop secure inter-domain routing with 
decent incentives for deployment 
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Recommendations (continued) 

5: Research incentives to improve resilience at the 

AS level 

6: Sponsor and promote good practice in network 

management 

7: Sponsor independent testing of routing 

equipment and protocols 

8: Conduct regular pan-European exercises and 

war games on the interconnection infrastructure 
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Recommendations (continued) 

9: Start figuring out the regulatory options 
in the event of transit market failure 

10: Promote public debate on policy and 
mechanisms for traffic prioritization 

1 1 : Work towards a resilience certification 
scheme so that corporate purchasers can 
identify dependable service providers, and 
to encourage deployment of BGPSEC etc 
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The Research Agenda 

• The online world and the physical world are 
merging; this will cause dislocation for years 

• Microeconomics can give us some of the basic 
tools we need to understand what's going on 

• The scientific challenge to understand how to 
manage the evolution of complex socio-technical 
systems 

• The business opportunity is to find ways to 
persuade ASes to provide the socially optimal 
level of excess capacity and do other necessary 
things to make the Internet secure and resilient 
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More . . . 

See www.ross-anderson.com for the report, and 

my security economics resource page 

And www.lightbluetouchpaper.org for our blog 

Workshop on Economics and Information 

Security (WEIS) - DC, June 201 1 

Workshop on Security and Human Behaviour 

(SHB)- DC, June 2011 

'Security Engineering - A Guide to Building 

Dependable Distributed Systems' 
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